Value Function Approximation and Policy Performance

Fig. 1 gives a geometric interpretation of value function approximation. We may think of J � as a vector in ∗; by considering approximations of the form J̃ = �r, we restrict attention to the hyperplane J = �r in the same space. Given a norm ≤ · ≤ (e.g., the Euclidean norm), an ideal value function approximation algorithm would choose r minimizing ≤J −�r≤; in other words, it would find the projection �r of J onto the hyperplane. Note that ≤J −�r≤ is a natural measure for the quality of the approximation architecture, since it is the best approximation error that can be attained by any algorithm given the choice of �. Algorithms for value function approximation found in the literature do not compute the projection �r , since this is an intractable problem. Building on the knowledge that J � satisfies Bellman’s equation, value function approximation typically involves adaptation of exact dynamic programming algorithms. For in­ stance, drawing inspiration from value iteration, one might consider the following approximate value iteration algorithm: �rk+1 = �T�rk, where � is a projection operator which maps T�rk back onto the hyperplane �r. Faced with the impossibility of computing the “best approximation” �r , a relevant question for any value function approximation algorithm A generating an approximation �rA is how large ≤J − �rA≤ is in comparison with ≤J − �r ≤. In particular, it would be desirable that, if the approximation architecture